Attachment

This article provides conceptual information for the ATTACHMENT data type. For instructions on how to work with an ATTACHMENT, see CRUD Operations > CREATE, READ, UPDATE, and DELETE.

Overview

The ATTACHMENT data type lets you link extensive amounts of binary data to a document for on-demand sync, both online or offline, without any conflicts. For instance, you can link and asynchronously sync large files that change infrequently or deeply embedded documents.

When incorporating an attachment into a document, rather than inserting the entire resource-heavy ATTACHMENT object, internally Ditto inserts a pointer object, known as the attachment token. (See ATTACHMENT Object and Attachment Token)

Unlike documents that are always accessible and automatically synced according to sync subscriptions, peers do not automatically fetch attachment data associated with a sync subscription. Therefore, before accessing an attachment, you must explicitly fetch it. (See Syncing ATTACHMENT Objects)

For a realworld usage scenario, see either the demo chat app for iOS or Android in the getditto > demoapp-chat GitHub repository.

ATTACHMENT Object

The ATTACHMENT object, shown on the right, stores your attachment outside the local device, is not directly visible in your code, and consists of metadata and a binary large object (blob) store.

• The metadata component is a set of string values providing extra details about the file or complex document, such as name and size, among other identifying information.
• The blob store is an array of bytes representing your large file or complex document encoded as an ATTACHMENT data type. (See Blob Store)

For example...

The following snippet in JSON demonstrates the structure of an ATTACHMENT object storing a JPEG file named “car_stock_photo.jpg”:

In this structure:

• The blob encodes the image.
• The metadata contains extra details about the binary contents, including its Multipurpose Internet Mail Extension (MIME) type and name as a set of string values.

Structure

{
    "blob": [...], // array of bytes
    "metadata": {
        "mime_type": "image/jpeg",
        "name": "car_stock_photo.jpg"
    }
}

MIME, also known as internet media type, is a standard identifier for specifying the structure and format of a document or file. It enables web browsers and other software systems to interpret and handle binary data appropriately when transmitting it over the internet. (See the official MDN Web Docs > MIME types (IANA media types))

Attachment Token

The attachment token, shown on the right, encapsulates information Ditto uses to identify and retrieve the full attachment when needed; as in, you’ve explicitly fetched it in your app.

For example...

Here is an attachment token structured within a document. In this structure, the picture attachment token acts as a placeholder for the full attachment storing a JPEG file named car_stock_photo.jpg:

Document

{
    "_id": "abc123",
    "make": "Hyundai",
    "year": 2018,
    "color": "black",
    "picture": {
        "id": "<unique identifier>",
        "metadata": {
            "mime_type": "image/jpeg",
            "name": "car_stock_photo.jpg",
        },
        "len": 123
    }
}

Schema

The following structure and corresponding table provide an overview of the schema the attachment token uses to model the data it contains:

JSON

{
  "id": "string",
  "len": 42,
  "metadata": {}
}

Here are the attachment token properties you can query to increase the efficiency of fetching operations in your app:

Field	Detail	Description
id	Attachment ID	A cryptographic hash of the attachment’s binary contents. (See Organizing by ID)
len	Blob Length	The size of the blob data in terms of length (len) in bytes. (See Blob Store)
metadata	Metadata	Additional information about the attachment, such as its name, type, and so on.

Blob Store

A JSON blob stores binary data representing any file type. For example, it can store image files like JPEG, PNG, and TIFF; video files such as MP4; audio files, including MP3 and WAV; document files like PDF; and more.

The metadata is stored in the document, while the blob is stored in a dedicated location independent of documents. The following table provides an overview of blob storage locations:

Type	Blob Store Location
Small Peer	- If running in the browser or a server-based system, in-memory storage. Specifically, within Random Access Memory (RAM). - If running on a mobile device like an iPhone, filesystem storage.
Big Peer	Cloud object storage service: Amazon WebServices (AWS) Simple Storage Service (S3).

Organizing by ID

Ditto organizes and stores blob data using the content-addressable network (CAN) paradigm. CAN is a distributed network architecture widely implemented in decentralized systems like Ditto’s where scalability, efficiency, and fault tolerance are essential requirements. (See the official Content-Addressable Network Wikipedia article.)

In more straightforward terms, Ditto organizes and accesses attachment data by the cryptographic hash table of its contents — for instance, attachment ID — rather than its physical location. A hash table is a data structure that stores key-value pairs.

By adhering to the CAN framework for attachment management, Ditto stores and retrieves data efficiently. For instance:

• Deduplication — If the same blob is created multiple times, Ditto stores only one copy. Similarly, if an attachment is added to multiple documents, they all reference the same blob.
• Conflict-free s**ync — Blobs are stored based on the cryptographic hash of their contents, ensuring that conflicts are effectively impossible.
• Decentralized — No central server is required to manage attachments. They can be created and accessed concurrently throughout the network, even with intermittent connectivity.

Purging Blob Data

Blob storage is managed internally using a reference-counting process known as garbage collection.

The garbage collection process running periodically in the background of Small Peers frees up space by removing blobs that no longer have any references — once an attachment has no remaining references, its blob is considered safe to remove, and the garbage collection process clears it from the device.

Syncing ATTACHMENT Objects

The two components comprising an ATTACHMENT object — metadata stored internally within a document and blob data storing the actual binary data of your attachment — sync in different ways:

• Metadata syncs just as any other document.
• Blob data syncs with an entirely different protocol than documents; one that is asynchronous and built on top of Ditto’s rainbow connection technology and optimizes for large binary data transfer. (See Mesh Networking 101)

Attachment blob sync being asynchronous provides three key advantages:

• Unblocking fetching operations so the document continues syncing without interruption when retrieving an attachment.
• Ditto multiplexes between these independent activities to balance needs for low latency and high throughput at the edge.
• Blob sync is resilient, so if connectivity is interrupted while transferring an attachment, progress is not lost. Once restored or if another peer possesses the attachment, Ditto resumes the transfer from the point of interruption.

Blob sync remains performant — Ditto uses data structures like Bloom filters to scale efficiently and remain light on system resources.

Bloom filters are randomized data structures designed to represent sets in a compressed form, making them highly effective in peer-to-peer environments where concerns such as memory usage and accuracy are prevalent. (See the official Bloom filter Wikipedia article.)

Garbage Collection

When an attachment token stored on a Small Peer end-user device is unused, meaning there are no remaining references to it, the garbage collection process running periodically in the background automatically deletes it to free up space.

Garbage collection is a reference-counting process that only runs on Small Peer devices (not on the Big Peer cloud deployment). It involves scanning for objects, including blob objects, that are no longer needed or relevant and are therefore safe to delete from its device.

The Big Peer stores its data in Ditto’s third-party cloud object storage service provider, Amazon WebServices (AWS) Simple Storage Service (S3). For more information, see the official Amazon Simple Storage Service Documentation.

Lazy-Load Retrieval Pattern

Since the blob representing your large file is stored outside the local Ditto store, fetch it only when needed. This on-demand retrieval pattern, known as lazy loading, reduces resource usage on end-user devices.

For a realworld usage scenario, see either the demo chat app for iOS or Android in the getditto > demoapp-chat GitHub repository. For instance, in the iOS demo chat app, you can see a savvy implementation of ATTACHMENT with a full-resolution avatar image from a collection named User.

Merge Strategy: Last-Write-Wins

The REGISTER and the ATTACHMENT type follow the last-write-wins principle to determine the value that ultimately takes precedence and merges across distributed peers. With this approach, each peer seeing the change will sequence it in the same order as every other peer, resulting in the same single value.

For example, one flight attendant updates a customer’s seat number to 6 and another to seat 9. When the two conflicting versions merge, the edit with the highest timestamp wins.

Put another way, by enforcing the last-write-wins merge strategy, for events A and B, where B is a result of A, event A always occurs before B.

Described as unsurprising and meaningful, the last-write-wins strategy means that, rather than arbitrarily resolving all conflicting changes to some predefined value, the value that results and propagates across peers as the single source of truth both reflects the actual end-user input and the update made by the last writer.

Creating and Working with Attachments

There are two separate steps in the process of creating an attachment:

1. Generate the attachment in the Ditto store. (See Initiating ATTACHMENT Objects)

2. Reference the attachment token in a document. (See Referencing Attachment Tokens)

To access your attachment data on demand or perform other operations, such as updating or deleting the attachment, first fetch its associated attachment token. (See Fetching Attachment Tokens)